Educational Media Evidence Rating and Technology Analysis Solution

ABSTRACT

A computer-implemented method includes receiving a comment to a medical study; prior to enabling storage of the comment in a data repository, transmitting, to a device used by a user who submitted the comment, a graphical user interface that prompts the user to enter an identifier that uniquely references a data repository record to a medical resource; receiving information indicative of the identifier that uniquely references the data repository record to the medical resource; verifying that the identifier references a data repository record to a medical resource; in response to verification of the identifier, storing the comment in association with the medical study in the data repository; and storing the identifier in association with the comment that is stored in the data repository, wherein the identifier provides evidence to support the comment.

CLAIM OF PRIORITY

This application claims priority under 35 U.S.C. §119(e) to provisionalU.S. Patent Application 61/726,759, filed on Nov. 15, 2012, the entirecontents of which are hereby incorporated by reference.

BACKGROUND

A medical database includes citations for medical literature from lifescience journals, online books, and so forth.

SUMMARY

In general, one aspect of the subject matter described in thisspecification includes receiving a comment to a medical study; prior toenabling storage of the comment in a data repository, transmitting, to adevice used by a user who submitted the comment, a graphical userinterface that prompts the user to enter an identifier that uniquelyreferences a data repository record to a medical resource; receivinginformation indicative of the identifier that uniquely references thedata repository record to the medical resource; verifying that theidentifier references a data repository record to a medical resource; inresponse to verification of the identifier, storing the comment inassociation with the medical study in the data repository; and storingthe identifier in association with the comment that is stored in thedata repository, wherein the identifier provides evidence to support thecomment. A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions.

The foregoing and other embodiments can each optionally include one ormore of the following features, alone or in combination. The actionsinclude storing a rule specifying that acceptance of a comment isconditioned on (i) submission of evidence and an identifier thatreferences the evidence, and (ii) on verification of validity of thereference; and responsive to receiving the comment, executing the rule;wherein transmitting the information for the graphical user interfacethat prompts the user to enter the identifier that uniquely references adata repository record to a medical resource comprises: responsive toexecuting the rule, transmitting the information for the graphical userinterface that prompts the user to enter the identifier that uniquelyreferences a data repository record to a medical resource. The actionsinclude identifying evidence that is referenced by the identifier;prompting users to provide information indicative of a quality of theevidence; and generating, based on the provided information, informationindicative of a rating of the evidence. The user submitting the commenthas a social connection in a social graph to a user who submitted themedical study. The actions include generating data for a graphical userinterface that when rendered on a display device displays a control,selection of which prompts the user to enter an identifier thatreferences a record that provides evidence for the comment. The actionsinclude preventing submission of the comment, prior to receipt of theidentifier. The actions include detecting that the comment received isunassociated with the identifier; wherein transmitting is in response todetecting. The actions include receiving, information indicative ofevidence that supports the comment. The actions include receivingevaluations of the evidence. The evidence comprises medical literature,and wherein the method further comprises: evaluating the medicalliterature based on the received evaluations.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of a system for educational media evidencerating and technology analysis.

FIGS. 2, 3A, 3B, and 4 are screen images of graphical user interfacesgenerated by the system.

FIG. 5 is a flow chart of processes executed by the system.

FIG. 6 is a block diagram of components of the system.

DETAILED DESCRIPTION

A system consistent with this disclosure collects data that isindicative of the collective efforts of physicians to evaluate medicalliterature databases (e.g., the medline database, the pubmed database,and so forth. Physicians use a physician social network to submit thisinformation.

FIG. 1 is a block diagram of an example of a network environment 100that enables an education media evidence rating and technology analysissolution. Network environment 100 includes network 106, client device104, server 108 and data repository 110. In the example of FIG. 1,client device 104 is used by a user, including, e.g., user 102. Networkenvironment 100 may include many thousands of data repositories, clientdevices, and servers, which are not shown.

In the example of FIG. 1, server 108 may be configured to implement aweb server, e.g., for providing web pages and/or web sites to clientdevice 104. In this example, server 108 generates study 112, e.g., amedical study based on questions and/or input received from users ofnetwork environment 100. In this example, user 102 wants to submitcomment 114 on study 112. In this example, prior to accepting submissionof comment 114, server 108 prompts user 102 to submit evidence thatsupports comment 114. Generally, evidence includes informationsupporting a statement.

In addition to prompting user 102 to submit the evidence, server 108also prompts user to submit an identifier that references the evidence.In an example, server 108 prevents or otherwise limits submission ofcomment 114, e.g., until user 102 has submitted the identifier. In thisexample, server 108 checks if user 102 has submitted an identifier. Ifserver 108 determines that the user has not submitted the identifier,server 108 continues to prompt user 102 to submit the identifier andonly allows submission of comment 114 once the identifier has beenentered.

Generally, an identifier includes a sequence of characters, numericvalues, and/or non-numeric symbols, or any combination thereof, thatuniquely identifies information and/or a set of identifier. There arevarious types of identifiers, including, e.g., a PubMed identifier(PMID). A PMID is a unique number assigned to a reference and/or arecord of a resource in a database, including, e.g., the MEDLINEdatabase of references and abstracts on life sciences and biomedicaltopics. A resource may include an item of data that is accessible from anetwork and/or a computing device. There are various types of resources,including, e.g., HTML pages, web pages, web sites, word processingdocuments, portable document format (PDF) documents, images, videos,audio files, electronic files, applications, medical literature andarticles and so forth.

In an example, the identifier is generated by an external database(e.g., the MEDLINE database). In this example, the identifier is knownto a user of the system, e.g., when the user reviews the resource in theMEDLINE database. In this example, the resource displays the identifierthat is associated with the resource. In still another example, theidentifier may be generated internally by the system, e.g., when thesystem stores resources.

In an example, server 108 implements a social network and/or a socialnetworking service. A social network is a social structure made up of aset of actors (such as individuals or organizations) and the dyadic tiesbetween these actors. In this example, user 102 is a user of the socialnetwork implemented by server 108. In this example, user 102 submitsstudy 112 (e.g., a question) to server 108. In this example, other usersthat are social connected to user 102 in the social network may submitcomments on the study, review the study, and so forth. Generally, asocial connection includes a link between nodes in a social graph,wherein a node represent a particular user.

Server 108 can be a variety of computing devices capable of receivingdata and running one or more services, including, e.g., an application,which can be accessed by client device 104. In an example, server 108can include a distributed computing system, a desktop computer, alaptop, a cell phone, a rack-mounted server, and the like. Server 108can be a single server or a group of servers that are at the sameposition or at different positions. Server 108 and client device 104 canexecute programs having a client-server relationship to each other.Although distinct modules are shown in FIG. 1, in some examples, clientand server programs can execute on the same device.

In the example of FIG. 1, client device 104 can be a variety ofcomputing devices capable of receiving data and running one or moreservices, including, e.g., an application. In an example, client device104 can include a server, a distributed computing system, a desktopcomputer, a laptop, a tablet, a smart phone, a cell phone, arack-mounted server, and the like.

Referring to FIG. 2, server 108 generates data for graphical userinterface 200, e.g., that may be rendered on client device 104. In thisexample, graphical user interface 200 includes portion 202 for displayof a study, including, e.g., a question that was submitted to server 108by a user of the network environment. Portion 202 is hereinafterreferred to as study 202, for purposes of convenience, and withoutlimitation. In this example, study 202 is associated with comment 204and evidence 206. In the example of FIG. 2, evidence 206 includes PMID208 that references a citation to the source of evidence 206. Evidence206 includes link 207, selection of which enables a user to view anarticle (or other literature) that supports the evidence or is otherwiserelevant to study 202. In an example, selection of link 207, causes adisplay (e.g., on a client device) of information (e.g., an article inthe PUBMED database) that is associated with PMID 208. In an example,selection of link 207 causes a user to access the PUBMED database.

Referring to FIG. 3A, server 108 generates data for graphical userinterface 300, e.g., that may be rendered on client device 104. Throughgraphical user interface 300, server 108 prompts a user to enter anidentifier for evidence 206 that is associated with comment 204. In thisexample, graphical user interface 300 includes entry field 302, e.g., afield for entry of an identifier. Graphical user interface also includeentry field 304, e.g., a field for entry of a summary of evidence 206.

In an example, server 108 is configured to verify the authenticity of anidentifier entered into entry field 302. In this example, server 108 maybe configured for communication with a data repository (e.g., datarepository 110 and/or the MEDLINE database) that stores identifiers. Inthis example, server 108 submits to the data repository the submittedidentifier along with a request to verify that the submitted identifieractually references a record in the data repository. If server 108receives, from the data repository, an indication that the submittedidentifier actually references a record in the data repository, server108 allows the user to submit the identifier and server 108 alsoassociates the identifier with the comment. If server 108 receives, fromthe data repository, an indication that the submitted identifieractually fails to reference a record in the data repository, server 108prevents the user from submitting the comment that was associated withthe identifier and prompts the user to submit a valid identifier.

In an example, data repository 110 may be configured to store variousidentifiers that reference various resources. In this example, server108 may be configured to periodically crawl the Internet and/or variousexternal data repositories to build a local repository of identifiers.In other examples, server 108 may be configured to access datarepositories that are external to network environment 100, e.g., inverifying the identifiers.

In still another example, the evidence is associated with theseidentifiers and the user needs to determine the identifier associatedwith a particular piece of evidence. In an example, a piece of evidenceis an item of medical literature. In this example, the item of medicalliterature references (e.g., displays) the identifier that is associatedwith the item of medical literature. By displaying the identifier, theuser may review the item of medical literature and identify theidentifier, e.g., to that the identifier may later be input when citingevidence (e.g., as a comment) for a study.

Referring to FIG. 3B, server 108 may also generate data for graphicaluser interface 350, e.g., which may be rendered by client device 104. Inthis example, graphical user interface 350 includes portion 352 fordisplay of a study and portion 354 for display of a comment that a userwants to submit for the study. In this example, portion 354 alsoincludes control 356, selection of which causes the user to be promptedto enter an identifier for a resource that provides evidence for thecomment. In an example, selection of control 356 causes a display ofgraphical user interface 300, e.g., entry of the identifier and of theevidence that is referenced by the identifier. In an example, thegraphical user interfaces displayed in FIGS. 3A and 3B enable a user toenter the types of information that are then displayed in FIG. 2.

In an example, hundreds and thousands of users review comment 204 andevidence 206 that is associated with comment 204. In this example,server 108 solicits, from the users, information regarding the qualityof evidence 206. Referring to FIG. 4, server 108 generates pollquestions 402, 404, 406 to collect quality information, including, e.g.,information that may be used in evaluating a strength and a quality ofevidence 206.

In this example, server 108 collects the quality information and usesthe quality information in rating the evidence, e.g., as shown inAppendix A. In an example, quality information for a particular piece ofevidence is used to evaluate (e.g., rate) that piece of evidence acrossfour different parameters, including, e.g., level of evidence, bias,clinical impact, and financial impact. In this example, qualityinformation for a particular parameter may be weighted, e.g., based onpredetermined weights. In this example, a particular item of evidencemay be assigned an overall quality value, e.g., based on quality values(some of which may be weighted) for the individual parameters.Generally, a quality value includes information indicative of a rating.A quality value for an individual parameter is based on the qualityinformation that is related to that parameter.

As shown in Appendix A, server 108 may be configured to compute a totalscore, a weighted score, and/or a raw score for a particular piece ofevidence, e.g., a particular resource that is associated with anidentifier. In an example, a total score includes a value indicate of atotal number of votes counted on poll. For example, a total score mayinclude a value indicative of a total number of responses to pollquestion 402. In this example, the total score may be for a particularparameter that is used in evaluating an item of evidence.

Server 108 may also be configured to generate a raw score. In thisexample, each answer to a poll question is associated with a value. Inthis example, the raw score is the average score for all the valuesassociated with answers to the poll questions. Server 108 may also beconfigured to generate a weighted score. In this example, the valueassociated with an answer to a poll question may be weighted, e.g.,based on an identity of a user who provides the answers. In an example,a user of server 108 may be associated with a user status, e.g.,information indicative of a level of experience and/or skill of a user.There are various types of user statuses, including, e.g., a status ofexpert, a status of student, a status of fellow, and so forth.

In this example, a user specifies a user status, e.g., when the usergenerates an account on server 108 and/or in the social network. In thisexample, a particular type of user status is associated with apredetermined weighted value. For example, answers that are provided byexperts may be weighted more heavily relative to the weight applied toanswers that are provided by users. In this example, the weighted scoreincudes the raw score where individual values that are used incalculating the raw score are weighted in accordance with a user statusof the user providing the answer that is associated with the value.

In an example, a particular piece of evidence may also be associatedwith clinical evidence, e.g., clinical results for a procedure relatedto the evidence. In this example, server 108 may poll users for qualityinformation related to the clinical evidence. Using the received qualityinformation related to the clinical evidence, server 108 generates aquality value for the clinical evidence, which may be combined with thequality values for the evidence that supports comments.

Referring to FIG. 5, server 108 implements process 400 in collecting andrating evidence. In operation, server 108 accesses (502) a study. Inthis example, server 108 also receives (504) a comment for the study.Server 108 also receives (506) evidence that supports the study. Uponreceipt of the comment and/or the evidence, server 108 determines (508)whether the evidence and/or the comment is associated with an identifier(e.g., a PMID). If server 108 determines that the evidence and/or thecomment is not associated with an identifier, then server 108 prompts(510) the user to enter an identifier that references the evidence.

If server 108 determines that the evidence and/or the comment isassociated with an identifier, then server 108 updates (512) datarepository 110 with the comment, the evidence and the identifier for theevidence.

In this example, following submission of the comment to server 108—wheresubmission of the comment is conditioned on a user who submits thecomment also submitting evidence and/or an identifier for the evidencethat supports a statement in the comment—server 108 may prompt otherusers to evaluate the evidence, e.g., by providing the other users withpolling questions. Based on answers to the polling questions, server 108rates (514) the evidence using the techniques described herein. In anexample, server 108 rates the evidence, e.g., by generating a raw scoreand/or a weighted score for a particular parameter that is indicative ofa level of quality of the evidence.

[I wouldn't use this one but rather would use a simple block diagram]FIG. 6 shows an example of computer device 600 and mobile computerdevice 650, which can be used with the techniques described here.Computing device 600 is intended to represent various forms of digitalcomputers, such as laptops, desktops, workstations, personal digitalassistants, servers, blade servers, mainframes, and other appropriatecomputers. Computing device 650 is intended to represent various formsof mobile devices, such as personal digital assistants, cellulartelephones, smartphones, and other similar computing devices. Thecomponents shown here, their connections and relationships, and theirfunctions, are meant to be examples only, and are not meant to limitimplementations of the techniques described and/or claimed in thisdocument.

Computing device 600 includes processor 602, memory 604, storage device606, high-speed interface 608 connecting to memory 604 and high-speedexpansion ports 610, and low speed interface 612 connecting to low speedbus 614 and storage device 606. Each of components 602, 604, 606, 608,610, and 612, are interconnected using various busses, and can bemounted on a common motherboard or in other manners as appropriate.Processor 602 can process instructions for execution within computingdevice 600, including instructions stored in memory 604 or on storagedevice 606 to display graphical data for a GUI on an externalinput/output device, such as display 616 coupled to high speed interface608. In other implementations, multiple processors and/or multiple busescan be used, as appropriate, along with multiple memories and types ofmemory. Also, multiple computing devices 600 can be connected, with eachdevice providing portions of the necessary operations (e.g., as a serverbank, a group of blade servers, or a multi-processor system).

Memory 604 stores data within computing device 600. In oneimplementation, memory 604 is a volatile memory unit or units. Inanother implementation, memory 604 is a non-volatile memory unit orunits. Memory 604 also can be another form of computer-readable medium,such as a magnetic or optical disk.

Storage device 606 is capable of providing mass storage for computingdevice 600. In one implementation, storage device 606 can be or containa computer-readable medium, such as a floppy disk device, a hard diskdevice, an optical disk device, or a tape device, a flash memory orother similar solid state memory device, or an array of devices,including devices in a storage area network or other configurations. Acomputer program product can be tangibly embodied in a data carrier. Thecomputer program product also can contain instructions that, whenexecuted, perform one or more methods, such as those described above.The data carrier is a computer- or machine-readable medium, such asmemory 604, storage device 606, memory on processor 602, and the like.

High-speed controller 608 manages bandwidth-intensive operations forcomputing device 600, while low speed controller 612 manages lowerbandwidth-intensive operations. Such allocation of functions is anexample only. In one implementation, high-speed controller 608 iscoupled to memory 604, display 616 (e.g., through a graphics processoror accelerator), and to high-speed expansion ports 610, which can acceptvarious expansion cards (not shown). In the implementation, low-speedcontroller 612 is coupled to storage device 606 and low-speed expansionport 614. The low-speed expansion port, which can include variouscommunication ports (e.g., USB, Bluetooth®, Ethernet, wirelessEthernet), can be coupled to one or more input/output devices, such as akeyboard, a pointing device, a scanner, or a networking device such as aswitch or router, e.g., through a network adapter.

Computing device 600 can be implemented in a number of different forms,as shown in the figure. For example, it can be implemented as standardserver 620, or multiple times in a group of such servers. It also can beimplemented as part of rack server system 624. In addition or as analternative, it can be implemented in a personal computer such as laptopcomputer 622. In some examples, components from computing device 600 canbe combined with other components in a mobile device (not shown), suchas device 650. Each of such devices can contain one or more of computingdevice 600, 650, and an entire system can be made up of multiplecomputing devices 600, 650 communicating with each other.

Computing device 650 includes processor 652, memory 664, an input/outputdevice such as display 654, communication interface 666, and transceiver668, among other components. Device 650 also can be provided with astorage device, such as a microdrive or other device, to provideadditional storage. Each of components 650, 652, 664, 654, 666, and 668,are interconnected using various buses, and several of the componentscan be mounted on a common motherboard or in other manners asappropriate.

Processor 652 can execute instructions within computing device 650,including instructions stored in memory 664. The processor can beimplemented as a chipset of chips that include separate and multipleanalog and digital processors. The processor can provide, for example,for coordination of the other components of device 650, such as controlof user interfaces, applications run by device 650, and wirelesscommunication by device 650.

Processor 652 can communicate with a user through control interface 658and display interface 656 coupled to display 654. Display 654 can be,for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) oran OLED (Organic Light Emitting Diode) display, or other appropriatedisplay technology. Display interface 656 can comprise appropriatecircuitry for driving display 654 to present graphical and other data toa user. Control interface 658 can receive commands from a user andconvert them for submission to processor 652. In addition, externalinterface 662 can communicate with processor 642, so as to enable neararea communication of device 650 with other devices. External interface662 can provide, for example, for wired communication in someimplementations, or for wireless communication in other implementations,and multiple interfaces also can be used.

Memory 664 stores data within computing device 650. Memory 664 can beimplemented as one or more of a computer-readable medium or media (e.g.,a machine-readable hardware storage device), a volatile memory unit orunits, or a non-volatile memory unit or units. Expansion memory 674 alsocan be provided and connected to device 650 through expansion interface672, which can include, for example, a SIMM (Single In Line MemoryModule) card interface. Such expansion memory 674 can provide extrastorage space for device 650, or also can store applications or otherdata for device 650. Specifically, expansion memory 674 can includeinstructions to carry out or supplement the processes described above,and can include secure data also. Thus, for example, expansion memory674 can be provide as a security module for device 650, and can beprogrammed with instructions that permit secure use of device 650. Inaddition, secure applications can be provided via the SIMM cards, alongwith additional data, such as placing identifying data on the SIMM cardin a non-hackable manner.

The memory can include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in an data carrier. The computer program productcontains instructions that, when executed, perform one or more methods,such as those described above. The data carrier is a computer- ormachine-readable medium, such as memory 664, expansion memory 674,and/or memory on processor 652, that can be received, for example, overtransceiver 668 or external interface 662.

Device 650 can communicate wirelessly through communication interface666, which can include digital signal processing circuitry wherenecessary. Communication interface 666 can provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication can occur, for example, through radio-frequencytransceiver 668. In addition, short-range communication can occur, suchas using a Bluetooth®, WiFi, or other such transceiver (not shown). Inaddition, GPS (Global Positioning System) receiver module 670 canprovide additional navigation- and location-related wireless data todevice 650, which can be used as appropriate by applications running ondevice 650.

Device 650 also can communicate audibly using audio codec 660, which canreceive spoken data from a user and convert it to usable digital data.Audio codec 660 can likewise generate audible sound for a user, such asthrough a speaker, e.g., in a handset of device 650. Such sound caninclude sound from voice telephone calls, can include recorded sound(e.g., voice messages, music files, and the like) and also can includesound generated by applications operating on device 650.

Computing device 650 can be implemented in a number of different forms,as shown in the figure. For example, it can be implemented as cellulartelephone 680. It also can be implemented as part of smartphone 682,personal digital assistant, or other similar mobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichcan be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms machine-readable medium andcomputer-readable medium refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying data to the user and a keyboard and a pointing device(e.g., a mouse or a trackball) by which the user can provide input tothe computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a user interface or a Web browser through which a user caninteract with an implementation of the systems and techniques describedhere), or any combination of such back end, middleware, or front endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication (e.g., a communicationnetwork). Examples of communication networks include a local areanetwork (LAN), a wide area network (WAN), and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

In some implementations, the engines described herein can be separated,combined or incorporated into a single or combined engine. The enginesdepicted in the figures are not intended to limit the systems describedhere to the software architectures shown in the figures.

For situations in which the systems and techniques discussed hereincollect personal information about users, the users may be provided withan opportunity to opt in/out of programs or features that may collectpersonal information (e.g., information about a user's preferences or auser's current location). In addition, certain data may be anonymized inone or more ways before it is stored or used, so that personallyidentifiable information is removed. For example, a user's identity maybe anonymized so that no personally identifiable information can bedetermined for the user, or a user's geographic location may begeneralized where location information is obtained (e.g., to a city, zipcode, or state level), so that a particular location of the user cannotbe determined.

All processes described herein and variations thereof (referred to as“the processes”) contain functionality to ensure that party privacy isprotected. To this end, the processes may be programmed to confirm thata user's membership in a social networking account is publicly knownbefore divulging, to another party, that the user is a member. Likewise,the processes may be programmed to confirm that information about aparty is publicly known before divulging that information to anotherparty, or even before incorporating that information into a socialgraph.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications can be made without departing fromthe spirit and scope of the processes and techniques described herein.In addition, the logic flows depicted in the figures do not require theparticular order shown, or sequential order, to achieve desirableresults. In addition, other steps can be provided, or steps can beeliminated, from the described flows, and other components can be addedto, or removed from, the described systems. Accordingly, otherembodiments are within the scope of the following claims.

What is claimed is:
 1. A method performed by one or more processingdevices, comprising: receiving a comment to a medical study; prior toenabling storage of the comment in a data repository, transmitting, to adevice used by a user who submitted the comment, a graphical userinterface that prompts the user to enter an identifier that uniquelyreferences a data repository record to a medical resource; receivinginformation indicative of the identifier that uniquely references thedata repository record to the medical resource; verifying that theidentifier references a data repository record to a medical resource; inresponse to verification of the identifier, storing the comment inassociation with the medical study in the data repository; and storingthe identifier in association with the comment that is stored in thedata repository, wherein the identifier provides evidence to support thecomment.
 2. The method of claim 1, further comprising: storing a rulespecifying that acceptance of a comment is conditioned on (i) submissionof evidence and an identifier that references the evidence, and (ii) onverification of validity of the reference; and responsive to receivingthe comment, executing the rule; wherein transmitting the informationfor the graphical user interface that prompts the user to enter theidentifier that uniquely references a data repository record to amedical resource comprises: responsive to executing the rule,transmitting the information for the graphical user interface thatprompts the user to enter the identifier that uniquely references a datarepository record to a medical resource.
 3. The method of claim 1,further comprising: identifying evidence that is referenced by theidentifier; prompting users to provide information indicative of aquality of the evidence; and generating, based on the providedinformation, information indicative of a rating of the evidence.
 4. Themethod of claim 1, wherein the user submitting the comment has a socialconnection in a social graph to a user who submitted the medical study.5. The method of claim 1, further comprising: generating data for agraphical user interface that when rendered on a display device displaysa control, selection of which prompts the user to enter an identifierthat references a record that provides evidence for the comment.
 6. Themethod of claim 1, further comprising: preventing submission of thecomment, prior to receipt of the identifier.
 7. The method of claim 1,further comprising: detecting that the comment received is unassociatedwith the identifier; wherein transmitting is in response to detecting.8. The method of claim 1, further comprising: receiving, informationindicative of evidence that supports the comment.
 9. The method of claim8, further comprising: receiving evaluations of the evidence.
 10. Themethod of claim 9, wherein the evidence comprises medical literature,and wherein the method further comprises: evaluating the medicalliterature based on the received evaluations.
 11. One or moremachine-readable hardware storage devices configured to storeinstructions that are executable by one or more processing devices toperform operations comprising: receiving a comment to a medical study;prior to enabling storage of the comment in a data repository,transmitting, to a device used by a user who submitted the comment,information for a graphical user interface that prompts the user toenter an identifier that uniquely references a data repository record toa medical resource; receiving information indicative of the identifierthat uniquely references the data repository record to the medicalresource; verifying that the identifier references a data repositoryrecord to a medical resource; in response to verification of theidentifier, storing the comment in association with the medical study inthe data repository; and storing the identifier in association with thecomment that is stored in the data repository, wherein the identifierprovides evidence to support the comment.
 12. The one or moremachine-readable hardware storage devices of claim 11, wherein theoperations further comprise: storing a rule specifying that acceptanceof a comment is conditioned on (i) submission of evidence and anidentifier that references the evidence, and (ii) on verification ofvalidity of the reference; and responsive to receiving the comment,executing the rule; wherein transmitting the information for thegraphical user interface that prompts the user to enter the identifierthat uniquely references a data repository record to a medical resourcecomprises: responsive to executing the rule, transmitting theinformation for the graphical user interface that prompts the user toenter the identifier that uniquely references a data repository recordto a medical resource.
 13. The one or more machine-readable hardwarestorage devices of claim 11, wherein the operations further comprise:identifying evidence that is referenced by the identifier; promptingusers to provide information indicative of a quality of the evidence;and generating, based on the provided information, informationindicative of a rating of the evidence.
 14. The one or moremachine-readable hardware storage devices of claim 11, wherein the usersubmitting the comment has a social connection in a social graph to auser who submitted the medical study.
 15. The one or moremachine-readable hardware storage devices of claim 11, wherein theoperations further comprise: generating data for a graphical userinterface that when rendered on a display device displays a control,selection of which prompts the user to enter an identifier thatreferences a record that provides evidence for the comment.
 16. The oneor more machine-readable hardware storage devices of claim 11, whereinthe operations further comprise: preventing submission of the comment,prior to receipt of the identifier.
 17. The one or more machine-readablehardware storage devices of claim 11, wherein the operations furthercomprise: detecting that the comment received is unassociated with theidentifier; wherein transmitting is in response to detecting.
 18. Theone or more machine-readable hardware storage devices of claim 11,wherein the operations further comprise: receiving, informationindicative of evidence that supports the comment.
 19. An electronicsystem comprising: one or more processing devices; and one or moremachine-readable media configured to store instructions that areexecutable by the one or more processing devices to perform operationscomprising: receiving a comment to a medical study; prior to enablingstorage of the comment in a data repository, transmitting, to a deviceused by a user who submitted the comment, information for a graphicaluser interface that prompts the user to enter an identifier thatuniquely references a data repository record to a medical resource;receiving information indicative of the identifier that uniquelyreferences the data repository record to the medical resource; verifyingthat the identifier references a data repository record to a medicalresource; in response to verification of the identifier, storing thecomment in association with the medical study in the data repository;and storing the identifier in association with the comment that isstored in the data repository, wherein the identifier provides evidenceto support the comment.